On the Statistical Comparison of Inductive Learning Methods

Experimental comparisons between statistical and machine learning methods appear with increasing frequency in the literature. However, there does not seem to be a consensus on how such a comparison is performed in a methodologically sound way. Especially the eeect of testing multiple hypotheses on the probability of producing a "false alarm" is often ignored. We transfer multiple comparison procedures from the statistical literature to the type of study discussed in this paper. These testing procedures take the number of tests performed into account, thereby controlling the probability of generating "false alarms". The multiple comparison procedures selected are illustrated on well-known regression and classiication data sets. 26.1 Introduction Recent interactions between the statistical and artiicial intelligence communities (see e.g. Han93, CO94]), have led to many studies that compare the performance of empirical statistical and machine learning methods on real-life data sets; examples are the StatLog We observe that there is no consensus in the research community on how such a comparative study is performed in a methodologically sound way. The ranking of k preselected methods is usually performed by training (estimating in statistical terminology) the methods on a single data set, and estimating their respective mean prediction errors (mpe) from a hold-out sample. The methods are subsequently ranked according to their estimated mpes. Some studies use, in our view appropriately, statistical signiicance testing in order to make this ranking. However, the eeect that comparing multiple methods has on the probability of generating a "false alarm" (claiming 1996 Springer-Verlag.